Technical  Report  1167 


Concept  Development  for  Future  Domains: 
A  New  Method  of  Knowledge  Elicitation 


Scott  B.  Shadrick  and  James  W.  Lussier 

U.S.  Army  Research  Institute 

Robin  Hinkle 

University  of  Louisville 

Consortium  Research  Fellows  Program 


20050829  028 


June  2005 


United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  unlimited 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 

A  Directorate  of  the  Department  of  the  Army 
Deputy  Chief  of  Staff,  G1 


ZITA  M.  SIMUTIS 
Director 


Technical  review  by 

Angela  Karrasch,  U.S.  Army  Research  Institute 
Paula  J.  Durlach,  U.S.  Army  Research  Institute 


NOTICES 

DISTRIBUTION:  Primary  distribution  of  this  Technical  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPE-ARI-MS, 

251 1  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 

FINAL  DISPOSITION:  This  Technical  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

NOTE:  The  findings  in  this  Technical  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT  DATE  (dd-mm-yy)  2.  REPORT  TYPE 
June  2005  Final 


4.  TITLE  AND  SUBTITLE 

Concept  Development  for  Future  Domains:  A  New 
Method  of  Knowledge  Elicitation 


6.  AUTHOR(S) 

Scott  B.  Shadrick,  James  W.  Lussier  (U.S.  Army  Research 
Institute),  and  Robin  Hinkle  (University  of  Louisville) 


3.  DATES  COVERED  (from. . .  to) 
March  2004  -  January  2005 


5a.  CONTRACT  OR  GRANT  NUMBER 


5b.  PROGRAM  ELEMENT  NUMBER 
622785 


5c.  PROJECT  NUMBER 
A790 


5d.  TASK  NUMBER 
211 


5e.  WORK  UNIT  NUMBER 


10.  MONITOR  ACRONYM 
ARI 


1 1 .  MONITOR  REPORT  NUMBER 
Technical  Report  1167 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  8.  PERFORMING  ORGANIZATION  REPORT 

ADDRESS(ES)  NUMBER 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

ATTN:  DAPE-ARI-IK 
2423  Morande  Street 
Fort  Knox,  KY  401 21  -5620 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND 
ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

251 1  Jefferson  Davis  Highway 
Arlington,  VA  22202-3926 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 
Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 
Subject  Matter  POC:  Scott  B.  Shadrick 


14.  ABSTRACT  (Maximum  200  words): 

During  the  development  of  operational  concepts  for  the  Future  Combat  System  of  Systems’  Unit  of  Employment  and 
Unit  of  Action  it  became  clear  that  the  Army  needed  a  more  effective  and  efficient  method  for  envisioning  the  future. 

A  review  of  existing  knowledge  elicitation  techniques  indicated  that  current  methods  are  not  easily  applied  to  the 
development  of  future  concepts.  They  do  not  adequately  address  the  cognitive  impact  new  concepts  and  technology 
have  on  Soldiers.  In  addition,  existing  methods  do  not  thoroughly  examine  the  potential  unforeseen  impacts  the 
introduction  of  the  new  technology  will  have.  The  inherent  difficulty  in  envisioning  new  concepts  requires  a  more 
systematic  approach  to  elicit  knowledge  from  domain  experts.  This  report  reviews  existing  methods  and  describes  a 
new  method  of  knowledge  elicitation  to  more  effectively  support  the  development  of  future  concepts,  evaluate  the 
impact  of  new  technology,  and  solve  difficult  problems  where  information  and  expertise  is  dispersed  among  many 
individuals. 


15.  SUBJECT  TERMS 

Concept  Development;  Knowledge  Elicitation;  Knowledge  Acquisition;  Cognitive  Task  Analysis 


SECURITY  CLASSIFICATION  OF 

16.  REPORT 
Unclassified 

17.  ABSTRACT 
Unclassified 

18.  THIS  PAGE 
Unclassified 

19.  LIMITATION 
OF  ABSTRACT 

Unlimited 


20.  NUMBER 
OF  PAGES 


21.  RESPONSIBLE  PERSON 
Ellen  Kinzer 

Technical  Publication  Specialist 
703-602-8047 


l 


Technical  Report  1167 


Concept  Development  for  Future  Domains: 
A  New  Method  of  Knowledge  Elicitation 


Scott  B.  Shadrick  and  James  W.  Lussier 

U.S.  Army  Research  Institute 

Robin  Hinkle 

University  of  Louisville 
Consortium  Research  Fellows  Program 


Armored  Forces  Research  Unit 
Barbara  A.  Black,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
251 1  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 


June  2005 


Army  Project  Number  Personnel  Performance  and 

622785A790  Training  Technology 

Approved  for  public  release;  distribution  unlimited 


iii 


FOREWORD 


As  new  technologies,  concepts,  and  doctrine  are  introduced  into  the  Army’s  operational 
systems  there  is  a  corresponding  change  to  the  cognitive  processes  and  procedures  used  to 
effectively  adapt  to  the  “new  way  of  accomplishing  a  task.”  A  serious  challenge  to  the  U.S. 

Army  transformation  effort  is  the  ability  to  develop  realistic  concepts,  systems,  and  technologies 
for  the  future.  Effective,  systematic  techniques  of  concept  development  and  knowledge 
exploration  for  envisioning  future  systems  and  operations  are  currently  not  available. 

Understanding  the  dynamic  interactions  between  people,  technology,  and  tasks; 
realistically  being  able  to  predict  new  technologies  and  their  impact  on  cognitive  processes;  and 
predicting  and  anticipating  potential  challenges  and  unintended  side  effects  will  be  key  enablers 
for  the  transformation  process.  All  too  frequently  new  technologies  are  introduced  without  even 
a  basic  understanding  of  these  relationships  and  the  resulting  systems  are  too  burdensome  to  the 
Soldier  or  far  too  cognitively  demanding  to  use  during  the  stress  of  battle.  This  report  reflects 
ongoing  work  by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 
as  part  of  its  Science  and  Technology  Objective:  IV.SP.2002.02,  Methods  and  Measures  of 
Commander-Centric  Training  to  develop  new  methods  to  enhance  the  U.S.  Army’s  ability  to 
produce  new  concepts  and  training  needed  to  support  the  transformation  effort.  The  techniques 
outlined  in  this  report  are  being  applied  to  ongoing  research  efforts. 

The  findings  of  this  research  have  been  briefed  to  representatives  of  the  Unit  of  Action 
Maneuver  Battle  Lab  (UAMBL),  to  Fort  Knox  officers  who  serve  as  Future  Force  Unit  of  Action 
Command  and  Staff  in  UAMBL  exercises,  to  the  Unit  of  Action  Experimental  Element,  and  to 
members  of  the  Lead  Systems  Integrator  Training  Integrated  Project  Team. 
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CONCEPT  DEVELOPMENT  FOR  FUTURE  DOMAINS:  A  NEW  METHOD  OF 
KNOWLEDGE  ELICITATION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  in  the  process  of  transforming  to  a  Future  Force  with  increased  reliance 
on  technology.  The  transformation  requires  the  development  of  new  concepts,  processes, 
procedures,  doctrine,  knowledge,  and  ideas.  During  the  development  of  operational  concepts  for 
the  (FCS)  Unit  of  Employment  and  Unit  of  Action  it  became  clear  that  the  Army  needed  a  more 
effective  and  efficient  method  for  envisioning  the  future.  Traditional  methods  of  concept 
development  and  knowledge  exploration  are  limited  in  their  ability  to  generate  realistic  and 
reliable  concepts  about  the  future.  While  an  impressive  list  of  candidate  technologies  and 
concepts  can  be  conceived,  the  Army  cannot  afford  to  become  too  optimistic  about  their 
influence  without  having  a  method  to  appropriately  evaluate  their  impact.  There  is  a  tremendous 
need  for  a  powerful,  yet  flexible,  approach  to  concept  development  and  knowledge  exploration 
that  can  cope  with  the  unique  difficulties  of  envisioning  and  studying  future  concepts. 

Procedure: 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  examined 
methods  to  generate,  refine,  test,  and  validate  new  concepts  related  to  doctrine,  tactics, 
techniques,  procedures,  and  technology  integration.  The  present  report  reviews  existing  methods 
of  concept  development  and  knowledge  exploration  and  analyzes  the  appropriateness  of  those 
methods  for  envisioning  future  concepts.  A  new  method  of  concept  development  and  knowledge 
elicitation  that  allows  for  the  systematic  investigation  of  future  concepts  is  proposed. 

Findings: 

The  current  research  concluded  that  existing  methods  of  concept  development  and 
knowledge  elicitation  are  not  appropriate  and  are  not  effective  for  examining  future  issues.  The 
inherent  difficulty  in  envisioning  new  concepts  is  explored  and  existing  methods  are  described. 

A  new  method,  incorporating  components  of  existing  methods,  was  developed  to  provide  a 
potentially  powerful  approach  to  systematically  investigate  future  concepts.  Information  on 
implementing  the  method  and  understanding  the  impact  of  future  concepts  and  technology  is 
provided. 

Utilization  and  Dissemination  of  Findings: 

The  findings  of  the  research  can  benefit  personnel  in  the  U.S.  Army  Training  and 
Doctrine  Command,  the  Unit  of  Action  Maneuver  Battle  Laboratory,  and  other  agencies 
involved  in  concept  development,  the  development  of  new  doctrine,  and  the  elicitation  of 


realistic  ideas  for  the  Future  Force.  Further,  the  outcomes  of  the  research  have  been  applied  to 
the  development  of  new  training  methods  for  the  U.S.  Army  and  the  U.S.  Army  National  Guard. 
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CONCEPT  DEVELOPMENT  FOR  FUTURE  DOMAINS: 
A  NEW  METHOD  OF  KNOWLEDGE  ELICITATION 


Introduction 

The  U.S.  Army  is  currently  in  the  process  of  transforming  into  a  strategically  responsive 
force  capable  of  rapid  deployment  and  effective  operations  in  all  types  of  military  missions. 
Under  the  transformation  effort,  the  Army  is  changing  the  way  it  equips,  organizes,  deploys,  and 
trains.  The  cornerstone  of  the  transformation  is  the  Future  Force  and  the  Future  Combat  System 
of  Systems  (FCS).  The  FCS  is  envisioned  to  be  a  distributed,  network-centric,  multi-mission 
combat  system  that  will  utilize  advanced,  state-of-the-art  technologies.  To  support  the 
transformation  effort  the  Army  is  developing  a  family  of  1 8  networked  war-fighting  systems 
which  are  more  lethal,  survivable,  deployable  and  sustainable  than  the  existing  arsenal  of  Army 
systems.  The  FCS  is  envisioned  to  be  equipped  with  a  combination  of  manned,  robotic,  and 
mixed  (man  and  robotic)  human-in-the-loop  equipment. 

It  is  predicted  that  the  network  of  systems  will  allow  unprecedented  information  transfer. 
The  extensive  amount  of  information  will  attempt  to  compensate  for  the  loss  of  armor  afforded 
by  current  systems  and  is  assumed  to  allow  for  superiority  and  synchronized  operations  to  see, 
engage,  and  destroy  the  enemy  before  the  enemy  detects  the  Future  Force.  It  is  predicted  that  the 
network  centric  approach  will  allow  for  an  improved  integration  of  communication  across  and 
within  echelons.  Further,  the  FCS,  and  its  accompanying  Future  Force,  is  envisioned  to  operate 
as  part  of  a  combined,  joint,  interagency,  and  multinational  team.  The  key  enablers  of  this  vision 
for  information  superiority  are  the  rapid  technological  advances  and  the  integration  of 
information  across  the  military.  One  of  the  most  challenging  problems  facing  the  Army  is  to 
determine  how  to  integrate  and  capitalize  on  future  capabilities  without  having  a  clear 
understanding  of  what  those  capabilities  are  or  how  they  should  be  implemented. 

The  standard  belief  that  the  introduction  of  new  technology  is  a  simple  process  that  will 
result  in  improved  organizational  performance  is  an  oversimplification  fallacy  (Feltovich,  Spiro, 
&  Coulson,  1997;  Woods  &  Dekker,  2000).  With  regard  to  technological  advancement  in  the 
U.S.  Army,  Wass  de  Czege  and  Biever  (1998)  highlight  this  point  suggesting  that  “combat 
power  is  not  simply  the  sum  of  machine  performance;  it  requires  individual  and  organizational 
competence  and  synergy”  (p.  19).  Thus,  a  change  in  technology  produces  a  corresponding 
change  in  the  operational  and  cognitive  systems  -  resulting  in  the  transformation  of  existing 
roles,  processes,  and  procedures  and  the  development  of  new  ones. 

Any  assertion  concerning  the  impact  of  future  technologies  represents  a  hypothesis 
regarding  the  relationship  and  dynamics  of  the  new  technology  to  people  and  existing  systems 
(Woods,  2002;  Woods  1998).  The  hypotheses,  once  observed  and  evaluated,  often  prove  to  be 
incorrect  once  the  second  order  effects  of  the  system  are  realized  (Sarter,  Woods,  &  Billings, 
1997).  Thus,  as  Woods  (2002)  suggested,  “envisioning  the  future  of  operations,  given  the 
dynamic  and  adaptive  nature  of  the  process,  is  quite  fragile”  (p.  2). 

Even  if  the  new  technology  is  successfully  integrated,  the  technology,  in  and  of  itself, 
does  not  allow  for  improved  performance.  In  fact,  the  insertion  of  new  technology  may  increase 
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cognitive  demands  (Howell  &  Cooke,  1989).  The  technology  must  be  coupled  with 
corresponding  changes  to  all  facets  of  the  system.  The  point  was  made  vividly  clear  by 
Cordesman  and  Wagner  (1996)  in  their  review  of  the  Gulf  War.  The  authors  stated, 

Much  of  the  equipment  deployed  in  U.S.,  other  Western,  and  Saudi  forces  was 
designed  to  ease  the  burden  on  the  operator,  reduce  fatigue,  and  simplify  the  task 
involved  in  combat.  Instead,  these  advances  were  used  to  demand  more  from  the 
operator. 

Almost  without  exception,  technology  did  not  meet  the  goal  of  unencumbering 
the  military  personnel  operating  the  equipment,  due  to  the  burden  placed  or-  them 
by  combat.  As  a  result,  weapons  and  support  systems  often  required  exceptional 
human  expertise,  commitment,  and  endurance.  The  Gulf  War  shows  that  there  is 
a  natural  synergy  between  tactics,  technology,  and  human  factors  and  effective 
military  leaders  will  exploit  every  new  advance  to  the  limit.  As  a  result,  virtually 
every  advance  in  ergonomics  was  exploited  to  ask  military  personnel  to  do  more, 
do  it  faster,  and  do  it  in  more  complex  ways. 

This  stress  occurred  whether  the  issue  was  flying  and  sustaining  higher  sortie 
rates,  conducting  faster  armored  maneuvers  over  longer  periods  of  time,  targeting 
and  firing  at  longer  ranges,  or  increasing  the  speed  of  logistic  support.  Wars  may 
occur  in  which  one  side  is  so  superior  that  it  can  rely  on  tactics  and  technology 
without  a  matching  improvement  in  manpower,  and  without  stressing  human 
factors  to  their  limit.  Such  wars,  however,  are  unlikely  to  involve  significant 
levels  of  conflict,  and  the  idea  that  technology  can  reduce  the  stress  of  combat  or 
the  need  for  manpower  quality  and  readiness  is  almost  certain  to  be  a  myth. 

While  there  is  no  way  to  prove  such  a  conclusion,  one  very  real  lesson  of  the  Gulf 
War  is  that  new  tactics  and  technology  simply  result  in  altering  the  pattern  of 
human  stress  to  achieve  a  new  intensity  and  tempo  of  combat,  (p.  25) 

The  future  of  U.S.  Army  dominance  will  not  continue  to  be  as  dependent  on  the 
Department  of  Defense  (DoD)  designing,  building,  and  protecting  unique  technologies.  Instead, 
dominance  will  be  maintained  by  designing,  implementing  and  protecting  the  world's  best 
technology  integration  process  (Carter  &  White,  2000).  This  is  a  different  way  of  doing 
business.  It  is  predicated  on  the  notion  that  the  U.S.  will  be  the  fastest  integrator  of  new 
technologies  -  into  the  hardware,  into  the  software,  into  our  doctrinal  and  organizational 
systems,  and  into  our  training  and  leader  development  process.  A  key  to  success  is  a  solid  Army 
process  for  concept  development  -  a  means  to  generate,  elaborate,  refine,  describe,  test,  and 
validate  new  Future  Force  concepts  relating  to  doctrine,  tactics,  techniques,  procedures,  unit  and 
team  organization,  job  allocation,  training,  leader  development,  and  other  aspects  of  technology 
integration. 

How  do  we  best  use  the  total  Army  team  -  warfighter,  scientist,  theorist,  analyst,  and 
engineer  -  in  the  development  process  of  future  concepts?  There  is  a  critical  need  to  create  the 
methods  needed  to  systematically  extract  critical  information  (knowledge  elicitation)  about  the 
future  issues  in  order  to  more  effectively  and  efficiently  manage  the  concept  development 
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(knowledge  creation)  and  integration  process.  This  report  addresses  that  need  by  reviewing 
existing  methods  and  documents  a  new,  systematic  method  of  knowledge  elicitation  to  address 
the  Army’s  need  for  a  formalized  process  for  concept  development. 

Importance  of  Knowledge  Elicitation 

Expert  knowledge  involves  the  reasoning,  judgment,  and  performance  of  individuals  who 
are  unusually  accurate,  skillful,  and  reliable  in  their  domain.  The  knowledge  is  mostly  heuristic, 
based  on  experience,  and  inaccessible  through  standard  observational  methods.  Knowledge 
elicitation  describes  a  process  in  which  a  researcher  or  analyst  captures  knowledge,  procedures, 
and  strategies  through  systematic  interaction  with  domain  experts  for  the  purposes  of  building 
expert  models  and  developing  expert  reasoning  strategies.  The  focus  on  expert  knowledge  has 
three  functions:  (a)  to  improve  and  support  human  performance  in  decision  making  tasks;  (b)  to 
design  instruction  to  efficiently  and  efficiently  train  individuals;  and  (c)  to  automate  task 
functions  that  consist  of  human  logic  processes. 

Traditional  methods  of  knowledge  elicitation  often  rely  on  direct  observation  of  expert 
behaviors.  The  outcomes  of  these  elicitation  efforts  can  be  decomposed  to  describe  knowledge 
that  is  linked  to  observable  behaviors  within  a  specific  domain  and  context.  In  addition,  other 
methods  have  been  developed  to  capture  internal  cognitive  processes  that  are  not  readily 
accessible  by  observing  behavior.  While  much  research  has  been  conducted  on  concept 
development  and  knowledge  elicitation,  the  acquisition  of  expert  knowledge  still  remains  one  of 
the  most  time-consuming,  tedious,  and  essential  tasks  in  designing  systems  (Chervinskaya  & 
Wasserman,  2000). 

Concept  development  for  the  Army  Future  Force  involves  future  systems  and  developing 
technological  capabilities.  It  is  concerned  with  future  operational  capabilities,  training,  and 
doctrine;  it  addresses  how  those  systems  should  be  developed  and  how  they  will  be  employed. 
When  the  goal  of  knowledge  elicitation  is  future  concept  development,  experts  are  required  to 
generate  best-guess  estimates  based  on  an  anticipated  set  of  new  capabilities.  In  other  words, 
while  the  participants  in  the  knowledge  elicitation  exercise  may  be  experts  in  their  current 
domain,  they  will  be  asked  to  theorize  and  reason  on  how  things  might  be  different  under  various 
projected  future  conditions.  How  does  knowledge  generalize  to  future  systems  where  detailed 
specifications  of  developing  technology  are  not  yet  known?  How  does  expertise  in  a  current 
domain  inform  the  development  of  concepts  for  future  products  and  practices? 

Current  Army  Methods  for  Concept  Development  and  Knowledge  Elicitation 

Generally,  the  Army  has  used  two  methods  to  elicit  new  knowledge  and  ideas  from 
experts.  The  first  is  to  have  a  strong  theorist  or  analyst  develop  and  present  new  concepts,  i.e., 
an  Army  thinker  thinks  and  then  writes.  A  good  example  of  this  is  Conceptual  Foundations  of  a 
Transformed  U.S.  Army  by  Wass  de  Czege  and  Sinnreich,  (2002).  While  strong  creative  thinkers 
are  clearly  a  vital  part  of  the  effort,  some  drawbacks  are  evident  in  the  ‘just  let  thinkers  think’ 
method  of  concept  development.  The  results  are  often  very  general,  have  not  benefited  from 
concrete  attempts  to  implement  the  ideas,  have  not  uncovered  difficulties  through  instantiation, 
can  be  hard  to  communicate,  and  do  not  easily  avail  a  team  of  specialized  contributors  working 
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jointly.  Several  efforts  less  successful  than  the  above-cited  example  could  be  mentioned  with 
regard  to  Future  Force  concepts,  producing  documents  that  are  poorly  thought  out,  too  general, 
impractical,  unrealistic,  fragmented,  disjointed  and  contradictory  ‘cut-and-paste’  efforts. 

A  second  Army  method  is  to  stand  up  a  replica  of  the  new  system  and  conduct  a  unit 
exercise  in  simulation,  such  as  a  focused  Combat  Training  Center  (CTC)  rotation  or  Advanced 
Warfighter  Experiment  (AWE).  Such  full-scale  events  can  be  effective  demonstrations  of 
moderate-to  well-developed  concepts  but  have  drawbacks,  especially  in  the  earlier  stages  of 
concept  formation.  They  are  expensive  and  require  great  effort  and  coordination  to  conduct,  do 
not  allow  flexible  manipulation  of  variables,  present  great  measurement  difficulties,  and  lack  the 
repetition  necessary  to  reach  well-founded  conclusions.  Further,  the  complexity  of  conducting 
the  event  tends  to  overwhelm  the  experimental  intent. 

A  limitation  of  both  approaches  is  that  they  often  do  not  account  for  the  realization  that 
the  new  concept,  technology,  or  process  will  likely  transform  what  it  means  to  be  an  expert  in  the 
field.  As  Woods  and  Dekker  (2000)  suggest,  “introducing  new  technology  is  not  manipulating  a 
single  variable,  but  a  change  that  reverberates  throughout  a  system  transforming  judgments, 
roles,  relationships,  and  weightings  on  different  goals”  (p.  5).  There  is  a  need  to  investigate 
intermediate  concept  development  and  exploration  methods  to  augment  the  two  extremes 
described  above,  methods  that  provide  structured  activities  to  measure,  assess,  and  guide  the 
concept  development  process  yet  are  flexible  enough  to  respond  rapidly  to  a  wide  range  of 
conceptual  constructions. 

Systematic  Methods  of  Analysis 

Systematic  methods  for  knowledge  acquisition  and  concept  development  are  available. 
These  methods  have  been  developed  based  on  cognitive  and  applied  behavioral  science 
principles  and  provide  structured  processes  by  which  critical  information  about  task  knowledge 
and  performance  can  be  identified. 


Methods  of  Task  Analysis 

Task  analyses  are  systematic  approaches  to  understanding,  describing,  and  defining  a  job 
in  terms  of  the  physical  and  cognitive  processes  needed  to  successfully  perform  particular  tasks. 
That  is,  task  analytic  methods  help  to  identify  the  knowledge,  skills,  abilities,  and  other 
characteristics  required  to  perform  a  job.  They  address  the  frequency,  difficulty,  criticality, 
conditions,  performance  standards,  and  importance  of  specific  tasks.  This  report  makes  a 
distinction  between  traditional  behavioral  approaches  to  task  analysis  and  cognitive  task  analysis. 

Traditional  task  analysis  focuses  on  the  way  a  task  is  performed  -  the  task  is  directly 
observable  with  a  clear  beginning  and  ending  point  (Department  of  Defense,  2001).  Several 
methods  have  been  used  to  gather  information  in  traditional  task  analytic  investigations.  The 
most  frequently  used  method  to  capture  task  information  is  through  the  use  of  individual  and/or 
group  interviews  with  subject-matter  experts.  Task  analysis  information  is  also  collected  using 
questionnaires  and  direct  observation  of  incumbent  subject-matter  experts. 
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McCormick,  Jeanneret,  and  Mecham  (1972)  make  a  distinction  between  job-  and  worker- 
oriented  methods  stating,  “The  job-oriented  concept  typically  would  be  reflected  by  the  use  of 
specific  task  statements  ...  the  worker-oriented  concepts  typically  would  be  reflected  by  the  use 
of  descriptions  of  reasonably  definitive  human  behaviors”  (p.  348). 

Job-oriented  task  analysis  focuses  on  what  the  task  involves  regarding  the  activities, 
tools,  products,  and  outcomes  of  the  task.  Job-oriented  task  analysis  is  a  systematic  process  for 
breaking  tasks  down  into  discrete  step-by-step  segments.  Typically,  this  is  accomplished  by 
interviewing  experts  and  by  observing  as  the  tasks  are  completed.  The  outcome  of  a  job-oriented 
task  analysis  is  a  list  of  tasks  and  subtasks  such  as  can  be  found  in  an  Army  Training  and 
Evaluation  Program  (ARTEP).  For  example,  the  task  Control  Tactical  Operations  is  subdivided 
into  eight  subtasks:  Manage  CS/CSS  Force  Positioning,  Manage  Use  and  Assignment  of 
Terrain,  Maintain  Synchronization,  Control  Tactical  Airspace,  Plan  Actions,  Sequels,  and 
Branches,  Make  Adjustments  to  Resources,  Concept  of  Operations,  or  Mission,  and 
Coordinate  Actions  to  Produce  Maximum  Effective  Application  of  Military  Power  (ARTEP  3- 
91 ,  Draft).  Techniques  classified  as  job-oriented  task  analysis  methods  include  the  Functional 
Job  Analysis  (Fine  &  Wiley,  1971),  Comprehensive  Occupational  Data  Analysis  Program 
(Christal  &  Weissmuller,  1976),  and  the  Task  Inventory  Analysis. 

Worker-oriented  task  analysis  focuses  on  requisite  characteristics  of  a  subject-matter 
expert  or  job  incumbent  expressed  in  terms  of  knowledge,  skills,  abilities,  and  other 
characteristics.  Worker-oriented  task  analysis  techniques  are  typically  used  to  identify  the 
activities  required  to  complete  a  task  and  does  not  specifically  identify  what  the  individual 
actually  does  (Burnett  &  McCracken  1982).  Techniques  classified  as  worker-oriented  task 
analysis  methods  include  the  Position  Analysis  Questionnaire  (McCormick,  Jeanneret,  & 
Mecham,  1972),  Critical  Incident  Techniques,  Occupational  Analysis  Inventory  (Cunningham, 
Boese,  Neeb,  &  Pass,  1983),  Common-Metric  System  (Harvey,  1993),  and  the  Job  Element 
Method  (Primoff  &  Eyde,  1988).  Figure  1  provides  a  more  comprehensive  list  of  task  analytic 
methods.  Please  see  Kirwan  &  Ainsworth  (1992)  and  Jonassen,  Hannum,  &  Tessmer  (1989)  for 
a  more  extensive  review  of  methods. 

Job-  and  worker-oriented  methods  are  very  effective  in  specifying  job  requirements, 
characteristics,  and  system  functions  for  observable  jobs  or  sets  of  task.  They  are  not,  however, 
effective  or  efficient  in  systematically  developing  an  understanding  of  cognitive  processes  or 
knowledge  requirements.  For  example,  the  tasks  and  subtasks  listed  above  from  Army  ARTEP 
3-91  do  not  indicate  anything  about  the  mental  processes  experts  go  through  to  perform  the  task 
well.  As  Gott  (1998)  stated,  “Complex  job  environments  require  deeper  cognitive  analyses  that 
can  ferret  out  the  conceptions  and  reasoning  processes  that  lurk  behind  observable  behaviors”  (p. 
11).  Cognitive  task  analysis  provides  a  systematic  process  for  identifying  the  cognitive  elements 
and  activities  needed  for  task  performance  and  is  useful  when  there  are  few  readily  observable 
behaviors.  The  focus  is  on  knowledge  and  cognitive  processes  as  opposed  to  a  breakdown  of 
procedural  steps. 
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Ability  Requirements  Scales 
Activity  Sampling 
Behavioral  Task  Analysis 
Checklist 

Common  Metric  Questionnaire 
Common-Metric  Systems 
Critical  Incident  Technique 
Decision-Action  Diagramming 
Decomposition  Method 
Fleishman  Job  Analysis  Survey 
Focus  Groups 

Functional  Analysis  System  Technique 
Functional  Job  Analysis  Scales 
Goal  Composition 
Groupware  Task  Analysis 
Heuristic  Task  Analysis  Process 
Integrated  Performance  Modeling 
Environment 
Interface  Survey 
Interviews 

Job  Components  Inventory 
Job  Element  Inventory 
Job  Elements  Method 
Link  Analysis 

Multipurpose  Occupational  Systems 
|  Analysis  Inventory-Closed  Ended 


Nominal  Group  Technique 
Observation 

Occupational  Analysis  Inventory 
Operational  Sequence  Diagrams 
Operator  Function  Model 
Petri-nets 

Position  Analysis  Questionnaire 
Procedural  Task  Analysis 
Questionnaires 
Scenario-Based  Analysis 
Sequential  Task  Analysis 
Signal  Flow  Analysis 
Table  Top  Analysis 
Task  Decomposition  Techniques 
Task  Inventory/Comprehensive 

Occupational  Data  Analysis  Programs 
Threshold  Traits  Analysis  (TTA) 
Timeline  Analysis 
Trigger  Analysis 
Verbal  Protocols 
Walk-Through  Analysis 
Work  Profiling  System  (WPS) 

Workload  Analysis 


Figure  1.  Listing  of  Task  Analysis  Methods. 

There  are  many  methods  of  cognitive  task  analysis,  each  with  unique  elements  designed 
for  various  settings  and  objectives.  All  the  methods  typically  involve  three  phases:  a  description 
of  the  task  environment,  identification  of  related  tasks  and  processes,  and  the  elicitation  of 
knowledge  (DuBois,  Shalin,  Levi,  &  Borman,  1995).  One  example  is  called  the  critical  decision 
method.  In  this  method  a  facilitator  asks  experts  to  recall  a  salient  incident  or  cases.  In  most 
situations,  the  case  identified  represent  non-routine  incidents  where  expert  performance  can  be 
highlighted.  The  incidents  become  the  focus  of  semi-structured  interviews  where  cognitive 
probes  are  used  to  understand  the  experts’  situation  assessment  and  decision  making  processes. 
Appendix  A  provides  a  brief  description  of  some  major  cognitive  task  analysis  techniques. 
Appendix  A  is  not  intended  to  be  an  exhaustive  presentation  of  such  methods.  Rather,  the 
examples  were  chosen  to  represent  the  variety  of  methods  currently  employed.  Figure  2 
provides  a  more  complete  list  of  methods  and  techniques.  Please  see  Schraagen,  Chipman,  & 
Shalin  (2000)  for  an  extensive  review  of  cognitive  task  analysis  methods. 
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Applied  Cognitive  Task  Analysis 

Active  Participation 

Activity  Sampling 

Behavioral  Descriptions 

Card  Sorting 

Cloze  Technique 

Cluster  Analysis 

Cognitive  Structure  Analysis 

Collect  Artifacts  of  Task  Performance 

Concept  Mapping 

Conceptual  Graph  Construction 

Critical  Decision  Method 

Critical  Incident  Method 

Critiquing 

Data  Flow  Modeling 

Decompose,  Network,  and  Assess  Method 

Decision  Analysis 

Diagramming 

Discourse  Analysis 

Distinguishing  Goals 

Dividing  the  Domain 

Document  Analysis 

Entity  Life  Modeling 

Entity-Relationship  Modeling 

Ethnographic  Methods 

Exploratory  Sequential  Data  Analysis 

Field  Observations 

Forward  Scenario  Simulation 

Goal  Decomposition 

Goal  Directed  Task  Analysis 

Goals,  Operators,  Methods,  Selection  Rules 

Group  Interview 

Hierarchical  Sort 

IDEF  Modeling 

Influence  Diagram  Construction 

Interesting  Cases 

Interruption  Analysis 

Interviewing  (Structured,  Unstructured, 

Semi-Structured) 

Laddered  Grid 
Likert  Scale  Elicitation 


Magnitude  Estimation 
Minimal  Scenario  Technique 
Multidimensional  Scaling 
Object  Oriented  Modeling 
Observation 
On-Site  Observation 
P  Sort 
Petri  Nets 

Precursor,  Action,  Results,  &  Interpretation 

Problem  Analysis 

Problem  Discussion 

Protocol  Analysis 

Proximity  Scaling 

Q  Sort 

Questionnaire 
Rapid  Prototyping 
Rating  and  Sorting  Tasks 
Reclassification 
Repertory  Grid 

Retrospective  Case  Description 
Role  Playing 
Semantic  Nets 
Shadowing  (Self/Others) 

Simulators  and  Mockups 
Step  Listing 
Storyboarding 

Structural  Analysis  Technique 

Stumbling  Block  Technique 

System  Examination 

System  Refinement 

System  Validation 

Table-Top  Analysis 

Task  Action  Mapping 

Task  Knowledge  Structures 

Teachback 

Think  Aloud 

Tutorial  Interview 

Twenty  Questions 

Uncertain  Information  Elicitation 

User  Needs  Analysis 

Wizard  of  Oz  Technique 


Figure  2.  Listing  of  Cognitive  Task  Analysis  Methods. 


With  its  concentration  on  non-observable  cognitive  processes,  cognitive  task  analysis  is 
more  amenable  to  the  investigation  of  future  systems  and  processes.  Specifically,  those  methods 
developed  for  the  knowledge  elicitation  phase  hold  considerable  promise  for  improving  our 
ability  to  create  new  concepts  and  develop  future  systems.  Components  of  various  cognitive 
methods  that  extend  traditional  approaches  for  describing  current  systems,  while  not  specifically 
designed  for  such  purposes,  can  be  used  to  systematically  capture,  analyze,  test,  and  evaluate 
future  constructs.  The  following  section  will  provide  more  detail  on  knowledge  elicitation 
techniques  and  methods. 

Knowledge  Elicitation 

Knowledge  elicitation  methods  are  a  key  component  of  the  information  gathering  stage  of 
task  or  cognitive-task  analysis  -  they  provide  the  mechanism  by  which  knowledge  is  transferred 
from  the  expert  to  the  analyst.  The  methods  attempt  to  provide  a  convenient  way  to  accurately 
and  easily  allow  experts  to  communicate  their  expertise.  Despite  the  abundance  of  elicitation 
techniques  available,  capturing  expert  knowledge  and  understanding  how  the  knowledge  relates 
to  problem  solving  capabilities  remains  an  obstacle  for  expert  system  development  (Moody, 
Blanton,  &  Will,  1998/1999).  Often,  the  knowledge  elicitation  used  to  develop  expert  systems  is 
vague  or  glossed  over  because  it  is  largely  ad  hoc  and  non-scientific  (Wright  &  Ayton,  1987). 
Klein,  Calderwood,  and  MacGregor  (1989a)  point  out  a  tendency  to  emphasize  explicit 
knowledge  (also  called  declarative  knowledge)  over  implicit  knowledge  (also  called  tacit 
knowledge).  Explicit  knowledge  is  information  that  is  easily  accessed,  expressed,  stored,  and 
applied.  Implicit  knowledge,  on  the  other  hand,  is  information  that  exists  in  the  experts’ 
unconscious  and  is  not  easily  accessible.  Experts,  through  years  of  practice,  develop  automatic 
habits  that  are  characteristic  of  expert  performance.  These  unconscious  chunks  of  knowledge 
usually  go  to  the  heart  of  what  makes  someone  an  expert.  Effective  knowledge  elicitation 
techniques  for  determining  the  unique  contributions  of  both  implicit  and  explicit  knowledge 
should  be  used.  Knowledge  elicitation  methods  that  overemphasize  explicit  knowledge  lead  to 
“the  mistaken  conclusion  that  explicit  knowledge  is  sufficient  for  performing  a  task  well”  (Klein, 
etal.,  1989a,  p.463). 

The  knowledge  elicitation  technique  should  provide  the  expert  with  an  appropriate  set  of 
procedures  for  transferring  information  to  the  researcher  or  analyst.  In  planning  knowledge 
elicitation,  there  are  several  decisions  that  must  be  made  to  ensure  that  the  process  is  effective 
and  the  results  are  reliable  and  valid  (Meyer  &  Booker,  1991).  The  researcher  must  determine 
the  setting  in  which  the  expert  judgment  is  to  be  gathered  and  select  a  form  of  communication 
for  asking  and  answering  questions.  The  elicitation  technique  chosen  determines  how  the 
experts  will  respond  and  how  their  responses  will  be  coded.  Often  it  is  important  to  allow  future 
participants  to  ‘see’  the  thought  processes  of  previous  experts.  Finally,  a  method  for  aggregating 
responses  from  multiple  experts  into  a  single  representation  is  needed  for  the  researcher  to 
communicate  and  describe  the  results. 

Communication  Techniques  and  Settings 

The  setting  describes  the  arrangement  of  the  meeting  between  researcher  and  experts. 
Several  broad  categories  have  been  used  to  effectively  elicit  knowledge  including  private 
interviews,  group  interviews,  interactive  group  discussions,  and  remote  response  reception. 
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These  settings  can  be  combined  or  modified  as  appropriate.  The  use  of  multiple  settings  may  be 
especially  important  when  collecting  observations  about  future  concepts  and  situations.  Detailed 
tacit  knowledge  can  be  collected  from  an  expert  from  a  specific  domain  area  during  a  private 
interview.  Group  interviews,  however,  allow  an  expert  from  one  domain  area  to  interact  with 
experts  from  other  domains.  Combining  the  perspective  from  different  domain  areas  may  be 
critical  for  the  analyst  to  truly  understand  goals,  technological  relationships,  interconnections, 
and  ideas  concerning  the  future. 

Private  interviews  should  be  semi-structured  and  can  be  used  to  develop  initial  concepts 
and  ideas  that  are  unimpeded  by  the  inherent  limitations  of  interactive  groups.  The  results  from 
the  private  interviews  feed  the  interactive  group  discussions;  including  face-to-face  meetings 
including  two  or  more  experts  from  multiple  domain  areas.  The  interactive  group  discussion 
technique  originated  in  marketing  as  the  focus  group  and  can  be  either  structured  or  unstructured. 
Meyer  &  Booker  (1991)  claim  that  interactive  groups  produce  a  larger  amount  of  data  with 
greater  predictive  value  than  do  individual  interviews  or  the  Delphi  method  (described  below) 
alone.  Feedback  from  other  participants  stimulates  conversation,  resulting  in  deeper  and  richer 
data.  A  possible  problem  with  this  method  is  groupthink,  a  frequent  tendency  of  groups  to  be 
less  critical  in  their  evaluation  and  not  question  poor  decisions.  Used  in  conjunction  with 
personal  interviews,  the  interactive  group  discussions  allow  expert  participants  to  debate  various 
topics  and  develop  a  consensus  by  aggregating  results. 

Elicitation  Techniques 

The  selection  of  a  knowledge  elicitation  method  for  a  specific  domain  involves  several 
considerations.  Criteria  for  selection  depend  upon  the  researcher’s  level  of  familiarity  with  the 
domain  and  the  amount  of  time  and  money  required  for  the  analysis.  Knowledge  elicitation 
methods  are  typically  divided  into  two  broad  categories,  direct  and  indirect.  Direct  methods 
involve  the  researcher  asking  domain  experts  to  tell  how  they  perform  their  job.  Examples 
include  the  various  forms  of  interviewing,  protocol  analysis,  simulation,  concept  mapping,  and 
the  Critical  Decision  Method  (CDM).  The  effectiveness  of  direct  methods  depends  upon  the 
articulation  and  cooperation  of  the  experts  and  their  ability  to  verbalize  the  information. 

Typically  experts  are  easily  able  to  verbalize  declarative  knowledge.  Declarative  knowledge, 
also  called  explicit  knowledge,  is  represented  by  classifications  and  relationships.  It  focuses  on 
descriptions  of  facts,  things,  methods,  or  procedures.  Although  facts  and  concepts  can  be 
expressed  verbally,  the  interview  process  may  not  always  yield  patterns  and  structures 
discemable  by  the  researcher  (Evans,  1988). 

Indirect  methods  are  thought  to  be  more  suitable  when  knowledge  is  not  easily  expressed 
by  the  expert.  Tacit  knowledge,  which  has  been  learned  implicitly  through  experience,  and 
overleamed  automatic  procedural  knowledge  can  often  be  difficult  for  experts  to  express. 

Indirect  methods  constrain  the  experts  to  state  their  knowledge  with  the  help  of  predefined 
structures  such  as,  repertory  grids  (Kelly,  1955;  Gaines  &  Shaw,  1992),  decision  trees,  card 
sorting  (Geiwitz,  Komell,  &  McCloskey,  1990;  Major,  1991),  and  laddering  techniques 
(Geiwitz,  et  al.,  1990;  Rugg,  Mcgeorge,  &  Shadbolt,  1990;  Cordingley,  1989).  It  is  worth 
noting,  however,  that  direct  methods  such  as  CDM  have  also  been  used  successfully  to  elicit  tacit 
knowledge.  In  addition,  indirect  methods  are  capable  of  complementing  direct  methods. 
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Due  to  the  complexity  of  most  domain  areas  and  the  difficulties  associated  with 
forecasting  future  capabilities  and  requirements  a  combination  of  several  methods  should  be 
employed  to  collect  information  from  knowledgeable  experts.  The  next  sections  will  briefly 
highlight  three  of  the  most  popular  methods  of  knowledge  elicitation.  As  with  the  review  of 
cognitive  task  analysis,  the  review  is  not  intended  to  be  exhaustive.  The  intent  is  to  provide  an 
overview  of  a  few  of  the  most  widely  used  methods  which  contribute  to  the  development  of  a 
new  method  of  elicitation  focused  on  the  elicitation  of  knowledge  for  use  with  future  situations 
and  concept  development. 

The  Delphi  Technique.  The  Delphi  technique  was  originally  developed  by  the  RAND 
Corporation  to  provide  a  method  to  elicit  expert  knowledge  and  develop  group  consensus  among 
participants  (Dalkey,  1969).  In  traditional  Delphi  approaches,  experts  do  not  interact  directly 
with  one  another  and  only  interact  with  the  researcher  in  a  limited  manner.  The  researcher 
gathers  responses,  makes  them  anonymous,  and  provides  them  to  other  expert  participants.  After 
reviewing  the  responses  provided  by  other  participants,  each  expert  submits  a  revised  response. 
The  process  is  repeated  as  many  times  as  necessary.  The  method  was  designed  to  avoid  the 
groupthink  bias  of  interactive  groups.  The  objective  of  bias  avoidance  is  met  if,  as  experts  revise 
their  answers,  they  are  converging  on  the  correct  answer.  Difficulty  arises  in  assessing  the 
quality  of  the  answer  and,  thus,  whether  groupthink  existed  in  reaching  consensus  (Meyer  & 
Booker,  1991).  Traditionally,  the  Delphi  method  used  the  standard  postal  mail  system  to 
distribute  and  collect  responses;  now,  however,  electronic-mail  is  usually  used  instead. 

Interviews.  Interviewing  involves  the  researcher  asking  the  expert  a  series  of  questions 
such  as  to  recall  and  describe  the  steps  taken  in  performing  a  task.  Sometimes  the  expert  is 
asked  to  differentiate  between  or  give  examples  of  terms,  concepts,  or  events.  Interviews  can  be 
unstructured,  structured,  or  semi-structured.  In  unstructured  interviews,  the  researcher  asks 
open-ended  questions  about  the  expert’s  reasoning  in  making  decisions  and  records  the 
responses.  This  interview  method  allows  the  researcher  to  become  familiar  with  jargon  and  gain 
an  overview  of  the  domain.  The  major  disadvantage  of  unstructured  interviews  is  that  disorder 
can  ensue  if  the  expert  speaks  off  topic  or  erroneously  assumes  that  the  researcher  is 
knowledgeable  in  the  domain  (Hoffman,  Shadbolt,  Burton,  &  Klein,  1 995).  Structured 
interviews  are  preplanned  to  reduce  the  time  length  of  the  interviewing  process  by  focusing  the 
expert  only  on  specific  questions.  Planning  structured  interviews  requires  that  the  researcher 
possess  some  knowledge  of  the  domain.  Semi-structured  interviews  are  often  the  preferred  style 
as  they  allow  the  researcher  to  add  questions  to  clarify  points  and  to  omit  questions  that  become 
redundant  or  irrelevant  as  the  interview  evolves.  The  technique  promotes  more  continuity  in  the 
data  than  unstructured  interviews.  Continuity  assists  in  the  comparison  and  aggregation  of 
responses  from  various  experts.  By  asking  the  expert  to  explain  the  rationale  or  assumptions  of 
the  response,  researchers  ensure  that  each  expert  is  responding  to  the  same  question.  Compared 
to  structured  and  unstructured  interviews,  semi-structured  interviews  are  more  likely  to  produce 
data  that  is  germane  without  the  inefficiencies  of  collecting  unnecessary  data  (Meyer  &  Booker, 
1991). 


Protocol  Analysis.  Protocol  analysis  involves  the  researcher  recording  the  expert  as  the 
task  is  completed.  Asking  the  expert  to  actually  perform  tasks  will  often  generate  more  valid 
knowledge  than  asking  experts  to  simply  describe  the  steps  required  by  asking  probing  questions 
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(Wright  &  Ayton,  1987).  In  addition,  the  expert  is  required  to  ‘think  aloud’  while  working 
through  the  problem  or  situation.  The  purpose  is  to  identify  knowledge  elements  and  steps 
required  for  problem  solving  in  the  given  domain  area.  The  method  has  the  advantage  of 
allowing  the  expert  to  perform  the  task  in  a  work  context  while  verbalizing  the  cognitive 
activities  required  for  success.  One  limitation  of  the  method  is  that  it  is  based  on  introspection 
and  bias  to  the  problem-solving  processes  (Wright  &  Ayton,  1987).  If  the  task  is  one  that  can  be 
performed  automatically,  then  experts  are  often  able  to  introspect  while  they  perform.  If, 
however,  the  task  has  a  substantial  cognitive  component,  the  think  aloud  requirement  may 
interfere  with  task  performance.  However,  this  limitation  can  be  negated  by  video  recording  the 
expert  during  task  performance  and  replaying  the  video  during  an  interview  session.  This 
process  is  called  retrospective  protocol  analysis  and  improves  the  amount  and  quality  of  tacit 
knowledge  elicited. 


Knowledge  Creation 

To  create  reliable  knowledge  it  is  not  sufficient  to  elicit  information  from  a  group  of 
domain  area  experts,  it  is  necessary  to  prod  them  to  go  beyond  what  they  already  know.  When 
developing  concepts  about  the  future,  researchers  should  conduct  a  thorough  review  of  current 
trends.  Three  methods  are  useful  in  this  regard:  environmental  scanning,  backcasting,  and 
technological  forecasting.  The  methods  produce  insights  that  guide  decision  making  and  are 
invaluable  to  the  development  of  realistic  scenarios  that  can  be  used  to  elicit  knowledge  and 
develop  new  concepts. 

Environmental  Scanning 

In  order  to  reliably  investigate  future  issues  and  scenarios,  a  firm  understanding  of  the 
current  environment  is  needed.  Environmental  scanning  is  a  systematic  method  of 
accomplishing  this.  In  this  context,  environmental  scanning  is  defined  as  the  acquisition, 
compilation,  synthesis,  analysis,  and  utilization  of  information,  knowledge,  trends,  and 
relationships  concerning  the  internal  and  external  environment  which  would  assist  in  selecting, 
planning,  and  implementing  a  realistic  course  of  action  (Aguilar,  1967).  Coates  (1985)  describes 
the  objectives  of  environmental  scanning  as:  detecting  scientific,  technical,  social,  political,  and 
economic  trends  relevant  to  the  problem  under  investigation,  defining  potential  threats, 
opportunities,  and  challenges  implied  by  those  trends,  enhancing  future-focused  thinking,  and 
identifying  trends  that  are  converging,  diverging,  impending,  distant,  or  interacting. 

Technological  Forecasting 

Technology  is  often  a  key  driver  in  the  development  of  any  new  product  or  system.  This 
is  particularly  the  case  where  military  systems  are  concerned.  The  ability  to  accurately  predict 
the  availability  of  a  given  technology  will  have  a  critical  impact  on  the  success  of  a  given 
program,  project,  or  system.  Technological  forecasting,  as  the  name  suggests,  is  interested  in 
forecasting  the  types  of  technologies  that  will  be  available  in  a  future  time  period,  the 
characteristics  of  those  technologies,  and  a  realistic  estimate  of  their  availability. 

There  are  two  primary  methods  of  technological  forecasting:  ontological  view  and 
teleological  view  (Frick,  1974).  The  ontological  view  assumes  that  technological  change  and 
innovation  are  the  result  of  scientific  and  technical  opportunities  and  advancement.  The 
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teleological  view  assumes  that  technology  change  and  innovation  are  the  result  of  environmental 
factors  (i.e.,  social,  economic,  political,  etc.)  that  demand  a  desired  end  result.  Thus, 
technological  forecasting  methods  can  be  classified  as  exploratory  (ontological  view)  or 
normative  (teleological  view).  Frick  (1974)  characterized  the  methods  by  stating  that  one  could 
think  of  exploratory  and  normative  forecasting  as  analysis  according  to  the  ‘push’  of 
opportunities  versus  the  ‘pull’  of  objectives.  Exploratory  methods  include:  economic  analysis, 
extrapolation  of  time-series,  learning  curves,  trend  analysis,  morphological  systematic 
exploration,  probabilistic  exploratory  forecasting,  qualitative  historical  analogy,  and  time- 
independent  contextual  mapping  (Jantsch,  1967;  Frick,  1974).  Normative  forecasting  methods 
include:  morphological  analysis  (schematic  and  matrix),  relevance  trees,  mission  flow,  decision 
matrices,  and  systems  analysis. 

Traditionally,  technological  forecasting  was  centered  on  the  estimates  by  recognized 
experts.  Those  estimates  may  no  longer  be  as  appropriate  since  technological  advancement  and 
progress  has  become  increasingly  more  dependent  on  the  unique  contributions  of  several  diverse 
technologies  (Meredith  &  Mantel,  2004).  Meredith  and  Mantel  (2004)  also  point  out  that  the 
degree  and  speed  of  technological  advancement  and  change  is  related  to  the  funding  of  several 
unique  technologies.  Therefore,  in  order  to  reliably  forecast  the  availability  of  future 
technological  systems  it  is  necessary  to  elicit  knowledge  from  a  range  of  experts  on  a  number  of 
relevant  topics. 

One  final  point  about  technological  forecasting  is  relevant  to  the  current  discussion.  The 
fact  that  technological  advancement  results  in  a  new  capability  does  not  mean  that  it  will  be  or 
should  be  put  to  immediate  use.  The  introduction  of  new  technology  or  technological  systems 
does  not  affect  a  single  variable  -  it  affects  and  transforms  the  entire  system,  transforming  roles, 
judgments,  goals,  and  relationships  (Carroll  &  Rosson,  1992).  The  successful  application  and 
implementation  of  a  new  technological  innovation  often  does  not  occur  until  five  to  seven  years 
after  the  discovery  (Rosegger,  1986).  It  takes  a  fair  amount  of  time  to  understand  how  to  most 
effectively  and  efficiently  implement  a  technology  while  understanding  the  effect  on  the  overall 
system  and  organization.  As  Harville  (2004)  suggested,  “it  is  also  possible  ...  to  pour  resources 
into  developing  a  new  technology  too  soon,  so  that  the  effort  is  largely  wasted  and  futile”  (p.  1). 

Backcasting 

Environmental  scanning  focuses  on  identifying  information  about  the  environment, 
including  the  current  state-of-the-art  and  likely  technological  advancements.  Backcasting  is  an 
extension  of  an  environmental  scan  to  identify  what  the  desired  future  should  look  like.  This  is 
in  contrast  to  forecasting,  which  attempts  to  predict  the  future.  As  Hojer  &  Mattson  (1999) 
suggest,  “forecasting,  based  more  or  less  on  prolonging  existing  trends,  is  of  little  value  beyond 
the  role  of  an  alarm-clock”  (p.  10). 

Backcasting  is  an  extension  of  a  thorough  and  reliable  forecast  that  begins  with  a 
desirable  future  endstate  (Robinson,  1982).  Once  the  endstate  or  desirable  future  has  been 
identified,  the  backcast  works  backwards  to  determine  what  reasonable  steps  must  be 
accomplished  to  reach  the  endstate  through  visionary  thinking.  The  backcast  should  include 
“what-if  ’  analysis  to  identify  conditions  and  risks  to  meeting  the  objective.  Thus,  the  backcast 
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can  help  identify  the  “technological  leaps”  required  to  reach  the  endstate.  The  backcasting 
approach  consist  of  six  steps:  determine  the  objectives,  specify  the  goals,  constraints,  and 
targets,  describe  current  state,  specify  exogenous  variable,  complete  scenario  analysis,  and 
complete  impact  analysis  (Robinson,  1990). 

Enabling  Knowledge  Creation 

When  looking  at  future  issues,  especially  issues  that  are  five  to  ten  or  more  years  out,  it  is 
necessary  to  move  from  elicitation  of  existing  knowledge  to  the  creation  of  new  knowledge. 
While  current  techniques  of  cognitive  task  analysis  and  knowledge  elicitation  provide  a 
mechanism  for  acquiring  existing  knowledge  they  are  not  designed  for  predicting  and  assessing 
realistic  observations  and  concepts  about  the  future.  Effective  methods  are  needed  for  capturing 
knowledge  from  available  information  sources  and  using  that  knowledge  to  create  new  ideas  and 
concepts. 

Based  on  the  distinction  between  explicit  and  tacit  knowledge,  Nonaka  &  Takeuchi 
(1995)  present  a  model  for  knowledge  creation  and  conversion.  Figure  3  reproduces  the  four 
patterns  of  interaction  between  explicit  and  tacit  knowledge  outlined  by  Nonaka  &  Takeuchi, 
1995).  The  four  quadrants  characterize  the  ways  in  which  knowledge  can  be  converted  into  new 
concepts  and  ideas.  The  authors  assert  that  tacit  knowledge  can  be  converted  into  new  tacit 
knowledge  through  the  process  of  socialization  -  the  sharing  of  experiences,  metal  models,  and 
technical  skills.  Tacit  knowledge  can  be  converted  into  new  explicit  knowledge  through  the 
process  of  extemalization  -  where  experts  interact  with  novices  and  transfer  information  using 
metaphors,  analogies,  and  models.  Explicit  knowledge  is  converted  to  tacit  knowledge  through 
the  process  of  internalization,  thought  experimentation,  and  individual  learning.  Finally,  explicit 
knowledge  can  be  converted  to  new  explicit  knowledge  through  the  process  of  combination.  In 
combination,  new  knowledge  is  created  by  reconfiguring  existing  knowledge  through  sorting, 
recategorizing,  or  other  methods  to  expand  the  understanding  of  existing  concepts. 
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Knowledge 

From 
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Figure  3.  Development  Process  for  New  Knowledge  and  Concepts. 
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In  sum,  knowledge  creation  is  a  continuous  process  for  transferring  and  sharing  tacit  and 
explicit  knowledge  with  groups  and  individuals  (Bloodgood  &  Salisbury,  2001).  The  knowledge 
creation  process  results  in  the  development  of  principles,  facts,  concepts,  procedures,  and 
processes  -  these  products  must  be  extended  to  develop  future  concepts.  The  modes  of 
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knowledge  creation  described  by  Nonaka  &  Takeuchi  seem  most  appropriate  when  transferring 
or  transforming  existing  knowledge  within  a  relatively  stable  system,  but  not  necessarily 
development  and  synthesis  of  unique,  unknown  concepts  and  ideas  for  a  U.S.  Army  future 
environment  that  is  changing  rapidly.  Figure  4  illustrates  an  expansion  of  the  process  to  include 
the  development  of  new  concepts.  The  critical  step  is  expanding  the  methodology  to  create 
unique  knowledge. 


Figure  4.  Concept  Development  Process. 

A  New  Method 

While  the  methods  described  above  provide  a  number  of  capabilities,  they  do  not  provide 
a  useful  or  suitable  process  for  the  development  and  evaluation  of  new  knowledge  and  concepts 
about  the  future.  A  new  method  is  needed  to  address  the  unique  aspects  associated  with  the 
development  of  new  concepts  and  prediction  of  future  realities.  This  section  will  describe  such  a 
method,  the  Flexible  Method  of  Cognitive  Task  Analysis  (FLEX). 

Although  the  FLEX  method  shares  many  features  with  other  methods  of  analysis  it  offers 
many  components  that  differentiate  it  from  traditional  methods  of  cognitive  task  analysis  and 
knowledge  elicitation.  Specifically,  it  focuses  on  the  development  and  evaluation  of  future 
knowledge.  The  FLEX  attempts  to  be  more  comprehensive  than  conventional  cognitive  task 
analysis  methods  by  attempting  to  capture  not  only  traditional  task  information  but  also  to 
provide  a  methodology  that  can  capture  the  conglomerate  of  interdependent  and  dynamic 
knowledge  sources.  The  method  allows  researchers  to  capture  existing  knowledge  and  it 
facilitates  the  creation  of  new  knowledge  and  concepts.  The  technique  is  similar  to  the 
information  acceleration  method  used  in  the  marketing  domain.  In  the  information  acceleration 
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method  where  companies  try  to  forecast  consumers’  response  to  new  products  by  providing  early 
models  to  focus  groups. 

The  FLEX  method  is  an  interview-based  problem  solving  approach  that  systematically 
develops  and  explores  future  concepts.  The  approach  uses  a  combination  of  the  Critical 
Decision  Method,  Protocol  Analysis,  the  Delphi  technique,  and  the  standard  interview.  Unlike 
existing  methods,  the  FLEX  method  grounds  the  experts  thinking  in  a  futuristic  setting.  Thus, 
knowledge  will  be  captured  by  employing  a  vignette-based  scenario  approach  where  experts  are 
provided  with  a  brief  multimedia  presentation  that  highlights  a  potential  future  situation  and 
requires  the  expert  to  solve  a  complex  problem  using  the  anticipated  capabilities.  To  do  so, 
experts  are  required  to  figure  out  how  to  solve  the  issues  presented  in  the  scenario  using  the 
resources  and  capabilities  provided. 

Similar  to  protocol  analysis,  participants  are  asked  to  verbalize  their  responses  by 
thinking  aloud.  However,  similar  to  the  Delphi  technique,  responses  from  each  participant  are 
provided  to  subsequent  participants.  Therefore,  subsequent  participants  are  able  to  identify 
weaknesses,  confirm  strengths,  and  build  upon  the  prior  responses.  Throughout  the  process,  a 
semi-structured  interview  is  used  to  probe  expert  knowledge  and  gain  a  deeper  understanding  of 
the  expert’s  reasoning.  Before  the  completion  of  the  process,  responses  from  subsequent 
participants  are  fed  back  to  the  originator  for  additional  input.  Finally,  a  small  group  of  experts 
are  used  for  interactive  group  discussions  allowing  for  consensus  building  and  validation. 

The  vignette-based,  scenario  approach  is  appropriate  in  this  context  because  it  is  often 
difficult  for  individuals  to  speculate  on  how  future  capabilities  might  be  employed.  Further, 
many  experts  tend  to  over-estimate  the  impact  of  future  technological  advances  and  to  ignore  the 
difficulties.  Thus,  forcing  the  participants  to  solve  a  concrete  problem  helps  ground  their 
thinking  so  that  more  appropriate  and  realistic  information  can  be  captured.  Figure  5  illustrates 
how  the  flex  process  produces  future  tactics,  techniques,  and  procedures  (TTP),  concepts,  and 
other  products.  The  following  section  summarizes  the  steps  involved  in  developing  a  FLEX 
scenario  and  conducting  a  knowledge  elicitation  and  concept  exercise. 

Phase  1 :  Domain  and  Problem  Identification 

In  the  first  phase  of  the  FLEX  approach,  it  is  necessary  to  define  the  domain  area  and 
identify  the  significant  problems  of  interest.  Defining  the  domain  will  help  specify  the  scope  of 
the  information  that  will  be  needed  and  will  assist  in  the  identification  of  experts.  Since  a  goal  of 
the  method  is  not  only  to  capture  new  knowledge  but  to  systematically  investigate  the  efficacy  of 
the  information  collected,  the  identification  of  a  significant  number  of  appropriate  experts  is 
required.  In  most  cases,  a  diverse  set  of  experts  will  be  required  from  a  variety  of  technical 
fields  -  particularly  in  cases  where  technological  advancements  are  expected  to  play  a  critical 
role  in  the  development  of  new  systems  and  processes.  During  the  problem  identification  phase, 
an  environmental  scan,  backcast,  and/or  technological  forecast  should  be  conducted  to  assist  in 
understanding  the  potential  for  technological  innovation  and  advancement. 
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Figure  5.  Overview  of  the  FLEX  Process. 

Phase  2:  Initial  Review  and  Analysis 

During  the  initial  review  and  analysis  it  is  important  to  capture  information  concerning 
the  current  state-of-the-art  for  the  domain  and  problem  space.  This  phase  is  similar  to  what 
would  be  expected  during  traditional  task  analysis  -  both  procedural  and  cognitive.  During  the 
initial  review  it  is  necessary  to  interview  experts,  document  current  processes,  and  gather 
information  about  expert  performance.  Contrary  to  traditional  task  analysis,  the  goal  is  not  to 
develop  a  complete  list  of  tasks  and  duties  in  a  given  domain.  The  goal  is  to  gain  an 
understanding  of  domain  expertise,  task  requirements,  processes,  and  outcomes.  During  the 
review  it  is  critical  to  identify  areas  where  innovation  may  lead  to  improved  performance  and 
processes  and  to  gain  an  understanding  of  current  research  and  immediate  technological 
innovation  for  the  domain. 

Phase  3:  Refine  Problem  Space  and  Develop  Initial  Scenario 

During  this  phase,  initial  decisions  regarding  the  knowledge  elicitation  scenario  should  be 
established.  An  initial  group  of  domain  experts  should  be  used  to  develop  one  or  more  scenarios 
with  appropriate  “branches.”  The  branches  should  represent  contentious  areas  where  multiple 
realities  are  possible.  The  scenarios  should  be  developed  in  an  iterative  fashion,  allowing  experts 
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to  develop  a  realistic  situation  capable  of  focusing  the  knowledge  elicitation  process.  At  this 
stage,  it  is  not  necessary  to  fully  develop  the  situation  -  it  is  only  necessary  to  establish  the  initial 
conditions  for  the  knowledge  elicitation  process. 

Phase  4:  Initial  Knowledge  Elicitation 

Based  on  the  scenario,  this  phase  should  elicit  knowledge  and  concepts  from  a  series  of 
experts  from  the  relevant  domain  areas.  The  scenario  should  be  used  as  a  starting  point  to  focus 
the  experts  on  a  particular  problem  area.  The  elicitation  process  should  allow  the  experts  to 
further  refine  the  scenario  by  adding  new  information,  challenging  existing  assumptions, 
anticipate  unintended  consequences,  and  predicting  technological  interventions.  During  the 
elicitation,  experts  should  be  asked  to  discover  new  ways  to  solve  problems  associated  with  the 
scenario  given  hypothesized  future  capabilities.  Knowledge  elicitation  should  focus  on  both 
individuals  and  small  groups  (e.g.,  dyads,  triads)  and  should  combine  experts  from  different 
specialty  areas. 

Phase  5:  Data  Reduction  and  Consensus  Building 

After  a  sufficient  sample  of  experts  have  had  the  opportunity  to  complete  the  knowledge 
elicitation  phases,  it  is  necessary  to  aggregate  the  data  into  common  and  meaningful  responses. 
The  data  reduction  phase  should  allow  a  small  group  of  experts,  a  different  group  from  phase  4, 
to  develop  a  consensus  on  the  efficacy  of  the  new  knowledge  and  concepts  captured  from  phase 
4.  The  information  can  then  be  used  to  update  the  scenario  to  reflect  the  new  knowledge  stream 
or  create  a  new  branch  to  highlight  “what  if’  situations.  The  altering  of  the  scenario  allows  the 
experts  to  “leave  their  fingerprints”  on  the  scenario  and  allows  an  iterative  process  for 
continuous  improvement. 

After  revising  the  scenario  based  on  the  initial  knowledge  elicitation  and  data  reduction 
phases,  it  is  necessary  to  reexamine  the  scenario  with  a  new  set  of  experts  and  provide  the 
revised  materials  to  former  participants.  This  process  provides  a  systematic  way  to  evaluate  the 
realism  of  the  new  information.  Thus,  Phases  4  and  5  may  be  repeated  several  times  until  a 
tested  solution  is  developed.  The  goal  of  the  iterative  process  is  to  develop  a  well  tested  and 
documented  solution  for  the  purpose  of  developing  new  theories,  principles,  tools,  techniques, 
and  procedures. 

Phase  6:  Knowledge  Representation  and  Concept  Documentation 

Knowledge  representation  provides  a  mechanism  for  documenting  and  displaying 
information  in  a  useable  format.  In  this  context,  knowledge  refers  to  organized  concepts, 
theories,  principles,  descriptions,  and  mental  models  of  descriptive,  procedural,  and  meta- 
cognitive  information.  The  goal  is  to  present  the  results  of  the  knowledge  elicitation  process  in  a 
meaningful  way.  Knowledge  can  be  represented  using  a  variety  of  methods,  including:  logic, 
semantic  networks,  production  rules,  frame-based  representations,  decision  trees,  graphs, 
diagrams,  charts,  and  tables. 
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Using  Experts  for  Knowledge  Elicitation  and  Creation 

The  previous  sections  of  this  document  have  outlined  the  use  of  experts  to  elicit 
knowledge  about  a  particular  task  or  domain  area.  Expertise  is  highly  specific  to  a  particular 
domain.  Cognitive  performance  is  interrelated  to  domain  content  and  context  (Newell  &  Simon, 
1972).  In  other  words,  experts  in  one  domain  area  may  be  novices  in  a  different  area. 

Experts  are  bound  by  their  history  and  experience  and  not  all  experts  are  able  to  reason 
beyond  the  information  and  knowledge  readily  available  in  their  environment.  When  confronted 
with  a  new  problem,  experts  call  upon  their  prior  knowledge  and  experiences  in  order  to  solve 
the  problem  or  predict  future  events.  There  is  no  guarantee  that  an  expert  in  an  existing  domain 
area  will  continue  to  be  an  expert  in  a  future  realization  of  that  domain.  In  research  leading  to 
the  development  of  the  FLEX  method,  some  of  the  military  subject  matter  experts  who  served  as 
participants  were  unable  to  free  their  thinking  from  current  doctrine.  Furthermore,  some  who 
have  spent  time  studying  the  Army’s  emerging  future  force  doctrinal  concepts  had  difficulty 
critically  evaluating  those  concepts.  Other  participants,  however,  were  stimulated  by  the  FLEX 
method  to  provide  extremely  unique,  creative,  and  well-reasoned  responses  (Gossman,  Mauzy, 
Heiden,  &  Flynn,  in  preparation).  It  did  not  seem  to  the  researchers  that  expertise  in  current 
operations  was  a  good  predictor  of  how  successful  the  participant  would  be.  Consider  the 
difficulty  administrative  assistants,  experts  in  using  typewriters,  had  adapting  to  the  introduction 
of  computers  to  the  workplace.  Yet,  these  experts  will  ultimately  contribute  to  and  influence  the 
decision  making  process  (Karat,  Karat,  &  Vergo,  2004)  regarding  future  systems. 

Often  it  is  assumed  that  a  domain’s  most  experienced  individuals  are  also  the  ones  with 
the  most  expertise.  Several  research  studies  have  questioned  the  accuracy  this  assumption  (e.g., 
Jacob,  Lys,  &  Neale,  1997;  Camerer  &  Johnson,  1991).  Many  of  these  studies  led  Ericsson 
(2000)  to  suggest,  . .  continued  improvements  (changes)  in  achievement  are  not  automatic 
consequences  of  more  experience  and  in  those  domains  where  performance  consistently 
increases  aspiring  experts  seek  out  particular  kinds  of  experience,  that  is  deliberate  practice.” 
While  experience  is  often  used  as  an  indicator  of  expertise  (e.g.,  hours  of  flying  time)  especially 
when  more  objective  measures  are  lacking,  it  tends  to  be  inadequate.  Experience  may  be 
necessary  for  expertise,  but  experience  is  not,  in  and  of  itself,  sufficient  in  the  development  of 
expertise  (Rohrbaugh  &  Shanteau,  1999). 

Even  if  true  experts  are  identified  there  is  a  concern  that  their  knowledge  may  be  biased. 
With  specific  attention  to  forecasting  future  technologies,  Tichy  (2002)  stated,  “top  experts  . .  . 
demand  a  more  active  policy  to  promote  their  field  of  work  and  tend  to  be  overoptimistic  with 
regard  to  the  realization  of  innovations”  (p.  19).  The  over-optimism  bias  is  most  notably 
expressed  in  terms  of  innovativeness  of  the  technology,  the  chance  of  realizing  the  technology, 
and  the  potential  exploitation  of  the  technology.  These  findings  led  the  author  to  conclude  that 
“foresight  exercises  should  include  not  only  top  experts  of  the  relevant  field,  but  also  experts 
with  a  broader  range  of  interest  as  well  as  experts  with  widely  differing  backgrounds”  (p.  4). 

The  failure  to  select  the  right  group  of  experts  in  concept  development  exercises  “is  likely  to 
bring  assessments  that  are  too  optimistic”  (Tichy,  2002,  p.  19).  In  the  Army  transformation 
planning  effort,  in  particular  the  development  of  Future  Combat  Systems,  initial  plans  were  very 
optimistic  and  remain  so  despite  several  reductions  in  requirements  and  increases  in  timelines. 
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The  question  remains  then,  how  do  we  identify  and  select  participants  for  an  elicitation 
exercise?  It  is  necessary  to  select  experts  that  are  capable  of  extending  their  knowledge,  mental 
models,  and  schemas  beyond  their  current  state  in  a  reliable  and  valid  way.  The  research  on 
identifying  appropriate  experts,  much  less  identifying  those  experts  capable  of  extending  their 
knowledge  to  future  concepts,  is  relatively  absent.  As  Subramani,  Peddibhotla,  and  Curley 
(2003)  suggested,  “Given  the  importance  of  the  issue,  it  is  surprising  that  it  has  received  little 
attention  in  prior  research”  (p.  4). 

Bloom’s  (1956)  taxonomy  provides  a  potential  starting  point  to  developing  a  systematic 
methodology  for  accurately  selecting  experts.  The  taxonomy  consists  of  six  categories 
(knowledge,  comprehension,  application,  analysis,  synthesis,  and  evaluation)  of  higher-order 
cognitive  thinking  skills.  Figure  6  illustrates  the  taxonomy  and  highlights  the  abilities  of  experts 
in  each  category.  Experts  at  the  synthesis  and  evaluation  level  may  have  the  required  abilities  to 
examine  future  issues  and  develop  new  knowledge  and  concepts.  These  abilities  include  the 
ability  to  role-play,  invent,  speculate,  and  develop  information  (synthesis).  Additionally,  these 
individuals  are  able  to  compare,  judge,  critique,  and  assess  concepts,  and  can  make  conclusions 
and  recommendations  based  on  information. 

Clearly,  identifying  appropriate  experts  for  an  elicitation  exercise  is  the  most  important 
component  of  the  process.  Future  research  efforts  must  focus  on  developing  techniques  and 
procedures  for  addressing  the  issue. 


Conclusions 

Potter,  Roth,  Woods,  and  Elm  (2000)  stated,  “in  performing  cognitive  task  analysis,  it  is 
important  to  utilize  a  balanced  suite  of  methods  that  enable  both  the  demands  of  the  domain  and 
the  knowledge  and  strategies  of  domain  expertise  to  be  captured  in  a  way  that  enables  a  clear 
identification  of  opportunities  for  improved  support”  (p.  321).  There  is  clearly  a  need  for  a 
systematic  approach  to  eliciting  information  needed  to  develop  future  concepts,  processes, 
systems,  and  procedures.  The  method  described  here  provides  a  flexible  method  for  eliciting  and 
creating  concepts  and  information  for  investigating,  creating,  testing,  and  understanding  future 
issues.  Ongoing  efforts,  such  as  the  U.S  Army  Research  Institute’s  project  to  develop  training 
for  crisis  action  planning  and  execution  (Beaubien,  2005),  are  using  the  method  to  evaluate  its 
effectiveness. 

In  concluding  this  analysis  it  is  important  to  restate  one  of  our  original  questions:  How 
do  we  best  use  the  total  Army  team  -  warfighter,  scientist,  theorist,  analyst,  and  engineer  -  in  the 
development  process  of  future  concepts?  Diaper  &  Stanton  (2004)  stated  that,  “Task  analysis 
without  psychology  is  like  peaches  and  cream  without  the  peaches:  thin  stuff’  (p.  567).  The 
same  can  be  stated  about  the  warfighter,  theorist,  analysis,  or  engineer.  The  Army  can  no  longer 
rely  on  haphazard  or  unsystematic  methods  for  concept  development  that  do  not  include  the  total 
Army  team.  Therefore,  the  critical  need  is  to  implement  a  process  that  systematically  allows  for 
effective  and  efficient  management  of  the  concept  development  process.  The  FLEX  method 
represents  an  attempt  to  integrate  and  exploit  the  capabilities  of  the  total  Army  team. 


19 


Figure  6.  Task  Oriented  Question  Construction  Wheel  Based  on  Bloom’s  Taxonomy,  ©  2004 
St.  Edward’s  University  Center  for  Teaching  Excellence. 
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Appendix  A 


Applied  Cognitive  Task  Analysis 

A  widely  used  approach  to  cognitive  task  analysis  is  Applied  Cognitive  Task  Analysis 
(ACTA).  The  ACTA  method  is  designed  to  provide  streamlined  techniques  to  elicit  and  capture 
the  cognitive  aspects  of  expert  performance  (Militello,  Hutton,  Pliske,  Knight,  &  Klein,  1997). 
The  intent  was  to  develop  tools  that  were  easier  to  use  than  the  methods  of  cognitive  task 
analysis  available  at  the  time  the  method  was  developed.  The  method  consists  of  three 
complementary  techniques  (task  diagram,  knowledge  audit,  and  simulation  interview)  used  to 
elicit  different  aspects  of  cognitive  performance  for  a  given  task.  The  objective  of  the  task 
diagram  is  to  provide  the  analyst  with  a  general  overview  of  the  domain  area  and  to  highlight  the 
cognitively  demanding  portions  of  the  task.  The  knowledge  audit  is  used  to  further  investigate 
the  specific  expertise  required  for  a  particular  task  or  subtask.  Finally,  the  simulation  interview 
uses  a  scenario  to  probe  the  cognitive  processes  involved  in  a  specific  situation.  The  results  of 
each  of  the  techniques  are  consolidated  in  a  cognitive  demands  table  used  to  synthesize  the  data 
so  that  it  can  be  applied  to  a  specific  situation  or  project.  The  ACTA  method  was  developed  to 
assist  instructional  designers  and  training  professionals  in  developing  training  that  focuses  on 
cognitive  skills. 

Critical  Decision  Method 

Using  the  Critical  Incident  Technique  originally  modeled  by  Flanagan  (1954),  Klein, 
Calderwood,  and  MacGregor  (1989b)  developed  a  technique  called  Critical  Decision  Method 
(CDM).  The  premise  of  the  CDM  is  that  useful  information  can  be  retrieved  by  having  experts 
recall  a  salient  incident  or  cases.  In  most  situations,  the  case  identified  represents  a  non-routine 
incident  where  expert  performance  can  be  highlighted.  The  incident  becomes  the  focus  of  a 
semi-structured  interview  where  cognitive  probes  are  used  to  understand  the  expert’s  situation 
assessment  and  decision  making. 

There  are  several  advantages  to  CDM.  The  semi-structured  format  is  flexible,  allowing 
the  interviewer  to  switch  to  more  relevant  issues  as  the  dialogue  progresses.  Focusing  on  critical 
incidents  highlights  the  very  capabilities  that  separate  experts  from  novices  allowing  little  time  to 
be  wasted  on  general  tasks  that  do  not  require  the  specialized  skill  of  the  expert.  Probed  recall  of 
such  tough  cases  is  effective  in  revealing  previously  inaccessible  knowledge  of  highly  skilled 
personnel.  As  Klein,  Calderwood,  &  Macgregor  (1989b)  state,  “the  focus  on  non-routine  cases 
also  makes  the  method  most  appropriate  to  eliciting  tacit  knowledge  that  is  not  part  of  the 
formalized  procedures  for  a  domain”  (p.  465).  The  categorization  of  CDM  as  a  specific  type  of 
technique  is  not  definitive  because  the  technique  incorporates  aspects  of  several  other  knowledge 
elicitation  methods.  Klein  (1996)  stated  that,  “the  CDM  is  a  protocol  analysis  method”  (p.  4). 
However,  CDM  lacks  the  ‘think  aloud’  aspect  of  standard  protocol  analysis  and  focuses  on  a 
prior  event,  rather  than  a  current  or  future  problem.  It  is  similar  to  a  case  study  in  that  a  scenario 
is  analyzed  from  a  decision  making  perspective,  but  in  CDM  the  expert  actually  experienced  the 
case  firsthand. 
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Precursor,  Action,  Results,  Interpretation  (PARI) 

The  PARI  technique  (Hall,  Gott,  &  Pokomy,  1 995)  is  a  specialized  interview  technique  that 
combines  the  technique  of  thinking  aloud  described  by  Ericsson  and  Simon  (1993)  with  expert 
and  novice  problem  solving  activity  (Gott,  1998).  The  Precursor,  Action,  Results,  Interpretation 
(PARI)  procedure  utilizes  a  “situated  problem-solving  session”  where  experts  demonstrate  their 
knowledge  based  on  a  specific  problem  or  context  while  being  probed  through  structured 
questioning  to  elicit  knowledge  and  expert  skills.  The  approach  attempts  to  identify  the  action  a 
performer  would  take  given  a  specific  precursor  event,  the  result  the  action  would  have  initiated, 
and  the  performer’s  interpretation  of  the  result.  The  approach  further  defines  the  cognitive 
process  by  allowing  the  performers  to  document  their  internal  mental  models. 

Decompose,  Network,  and  Assess  Method  (DNA) 

The  DNA  method  was  developed  to  provide  a  practical  approach  to  cognitive  task 
analysis  that  was  capable  of  eliciting  and  capturing  task  information  from  experts  and  novices 
across  a  wide  range  of  domain  areas  (Shute  &  Torreano,  1995).  The  DNA  method  is  supported 
with  software  tools  that  attempt  to  automate  the  knowledge  elicitation,  knowledge  organization, 
and  management,  and  hierarchically  structure  knowledge  for  the  given  domain.  Shute,  Torreano, 
and  Willis  (1999)  describe  the  DNA  as  a  method  to  “elicit  knowledge  from  subject-matter 
experts  which  will  Decompose  a  domain,  Network  the  knowledge  into  comprehensive  structures, 
and  employ  other  experts  in  a  given  domain  to  Assess  the  validity,  completeness,  and  reliability 
of  the  knowledge  structures”  (p.  369). 

The  method  and  software  consist  of  four  modules  (Customize,  Decompose,  Network,  and 
Assess).  The  customize  module  is  used  to  provide  the  analyst  with  a  means  to  provide  specific 
information  about  the  domain  area.  The  decompose  module  operates  as  an  interactive,  semi- 
structured  interview  and  is  used  to  elicit  procedural  and  conceptual  domain  knowledge  from 
subject-matter  experts.  The  intent  of  the  network  module  is  to  transform  expert  knowledge  into 
knowledge  hierarchies,  graphs,  and  rules.  The  assess  module  is  used  to  validate  the  information 
acquired  and  products  developed  during  the  elicitation  process.  Here,  multiple  experts  are 
“employed  to  review  and  edit  one  another’s  conceptual  graphs  as  a  method  of  validating 
externalized  knowledge  structures”  (Shute,  Torreano,  &  Willis,  1999,  p.  375). 

Task  Knowledge  Structures  (TKS) 

The  TKS  method  was  derived  from  work  by  Johnson,  Diaper,  and  Long  (1984)  on  Task 
Analysis  for  Knowledge  Descriptions  and  attempts  to  identify  the  various  aspects  of  knowledge 
and  information  requirements  for  a  task.  The  TKS  method  characterizes  the  knowledge  of 
subject-matter  experts  based  on  previous  experiences.  A  major  assumption  of  TKS  is  that  during 
the  development  of  expertise  individuals  develop  specific  knowledge  structures  (Hamilton, 

1996)  that  are  represented  in  memory  (Johnson  &  Johnson,  1991).  The  TKS  method  provides  a 
mechanism  for  collecting  information  about  how  experts  perform  required  tasks  and  consists  of 
two  separate  components,  a  goal  structure  and  a  taxonomic  structure.  The  goal  structure 
identifies  the  task  activities,  goals,  subgoals,  and  provides  an  understanding  of  the  steps  needed 
to  complete  the  task.  The  taxonomic  structure  includes  information  about  actions  and  objects, 
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including  their  classification,  attributes,  relationships  to  one  another,  features,  and  typicality 
rating  (Johnson  &  Johnson,  1991).  The  TKS  method  assumes  that  the  task  knowledge  is 
structured  in  a  meaningful  way  and  can  be  analyzed,  modeled,  and  predicted. 

Goals,  Operators,  Methods,  Selection  Rules  (GOMS) 

The  GOMS  method  (Card,  Moran,  &  Newell,  1983)  is  a  cognitive  task  analysis  and 
modeling  method  that  has  been  used  extensively  in  the  analysis  of  human-computer-interactions 
to  document  expert  performance  models.  The  approach  is  composed  of  four  parts:  goals, 
operators,  methods,  and  selection  rules.  Goals  are  considered  to  be  the  endstate  of  the  task  when 
successfully  completed  -  what  the  user  wants  to  achieve.  Operators  are  the  procedural, 
perceptual,  psychomotor,  and  cognitive  actions  that  must  be  taken  to  accomplish  the  various 
goals  and  subgoals.  The  operators  are  sequenced  to  develop  the  methods  or  the  specific  step-by- 
step  procedures  for  accomplishing  the  tasks.  Operators  are  assigned  an  execution  time 
represented  the  average  amount  of  time  required  to  complete  the  steps.  Selection  rules  consist  of 
the  “if-then”  decision  rules  and  conditions  that  experts  used  to  determine  which  method  should 
be  used  to  complete  a  particular  goal  or  subgoal. 

The  GOMS  method  is  based  on  the  assumption  that  experts  interacting  with  a  system  are 
involved  in  problem  solving.  By  analyzing  how  experts  interact  with  the  system  it  is  possible  to 
decompose  the  task  or  problem  into  subproblems  and  an  understanding  of  the  goals  and 
expectations  can  be  developed  allowing  for  a  logical  sequence  of  operations  and  timing  values 
for  each  operation  to  be  documented.  The  model  also  provides  a  mechanism  for  making 
predications  about  how  experts  will  perform  given  a  proposed  design.  The  method  is  not  well 
suited  for  complex  cognitive  skills  or  for  use  with  novices.  However,  there  have  been  several 
variants  to  the  GOMS  method  to  address  many  of  the  methods  shortcomings.  The  Natural 
Goals,  Operators,  Methods,  and  Selection  Rules  Language  (NGOMS,  Kieras,  1988)  method,  for 
example,  added  a  mechanism  to  deal  with  expert-novice  differences. 

Goal-Directed  Task  Analysis  (GDTA) 

The  GDTA  method  is  a  cognitive  task  analysis  technique  that  concentrates  on  identifying 
an  expert’s  situational  awareness  needs  and  requirements  for  a  particular  task  (Endsley,  1993). 
Endsley  (2000)  defined  situational  awareness  as  those  “dynamic  information  needs  associated 
with  the  major  goals  or  subgoals  of  the  operator  in  performing  his  or  her  job  (as  opposed  to  more 
static  knowledge  such  as  rules,  procedures,  and  general  system  knowledge”  (p.  149).  Situational 
awareness  requirements  can  be  described  by  three  levels  of  awareness.  Level  1  Situational 
Awareness  -  Perception,  involves  the  perception  and  identification  of  significant  factors  in  the 
environment.  Level  2  Situational  Awareness  -  Comprehension,  requires  the  expert  or  operator 
to  interpret,  store,  retain,  integrate  multiple  pieces  of  information,  and  understand  the  relevance 
to  obtaining  a  particular  goal.  Level  3  Situational  Awareness  —  Projection  requires  the  ability  to 
predict  future  events  -  the  highest  level  of  awareness. 

Thus,  the  GDTA  method  systematically  identifies  the  situational  requirements  needed  for 
effective  decision  making.  The  actual  analysis  is  conducted  using  a  mixture  of  approaches, 
including:  observation,  verbal  protocol  analysis,  questionnaires,  analysis  of  written  materials. 
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and  various  knowledge  elicitation  techniques.  The  GDTA  method  has  been  successfully  applied 
to  a  wide  variety  of  domains  (Endsley,  1 999). 
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